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1. INTRODUCTION 

Rosman et al. [1] define referencing competency as the combination of three distinct factors: 
knowledge, skills, and ability of an individual that are particularly related to using, editing, and disseminating 
various types of reference standards as well as referencing software. Competency in referencing is very 
important and considered as among the 21st-century skills-especially related to critical thinking and problem- 
solving skills [2]. As such, universities were challenged to rethink their strategies and educational practices 
beyond the traditional approach [2]. Subsequently, Chan and Yeung [3] argue the importance of holistic 
competencies and the challenges to embracing such competencies in the universities. 

In the academic context, postgraduate education is the highest level of education that consist of 
doctoral and master education. Out of the two types of education level, doctoral education is much more 
challenging and require specific contribution for passing the evaluation, such as the total number of 
publication, and innovation awards and some universities do require their postgraduate student to publish a 
paper in a reputable journal as part of the completion criteria [1], [4], [5]. Thus, one of the issues that usually 
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deterred the probability of publishing a paper among postgraduate students is the quality of the manuscript 
produced. As known, a good quality paper was produced out of the good quality of references, such as 
referring to reputable online databases, reputable journals, and peer-reviewed articles [5]. Henceforth, 
knowledge on referencing competency influence individual performance and enhance the probability of 
getting published [1]. As such, many students and academicians would fall into the trap of predatory journals 
as a shortcut of publishing their work, especially in the wake of Novel Coronavirus 2019 (COVID-19) [6], 
[7]. Succumbed to the predatory journal would jeopardize the reputation and work quality of any individual, 
regardless of academician or postgraduate students [7]. 

Beninger et al. [8] define predatory journals as exploiting the norm of the scientific community by 
producing or publishing manuscripts without proper scientific evidence. More companies have since emerged 
offering fast track publication and easy publication into this kind of fraudulent journal [9]. Over the past 
decade, the number of predatory journals increase exponential [7]—[9]. Gill [7] lists publication fee, peer 
review, publication speed, failure to follow publishing standards, poor quality control, and poor verification 
of submission authenticity, inappropriate intellectual property protection, and falsified editorial board as the 
major indicators of a fraudulent journal. However, despite the revelation of the negative impact of predatory 
journals, some scholars simply ignored this warning; mostly due to focussing on adding a new publication, 
desire for quick publication turnaround, and lack of knowledge on false indexing provided by the journal and 
flattering email [10]. 

Thus, previous similar studies on referencing competency suggested that one of the challenging 
issues in the publication is to ensure the quality of the references and citation. However, the lack of mobile 
applications available that focus on citation management caused several problems such as succumbing to 
predatory journals, poor citation work, desk rejection, and inaccurate facts. As a result, an application system 
CiteGuru was developed based on hypertext pre-processor (PHP) and my structured query language 
(MySQL) database to help academician and postgraduate students to deter the influence of predatory journals 
and enhance the quality of manuscripts submitted to a reputable journal. Henceforth, the purpose of this study 
is twofold: first, to investigate the roles and usability assessment of CiteGuru application as a mobile solution 
for improving learners’ skills, ability, and knowledge on referencing and second, to investigate the roles of 
demographic profiles on the usability perception among the respondents. 


2. CITEGURU APPLICATION 

CiteGuru is a mobile-friendly web application system that focused on the American Psychological 
Association (APA) 7" edition referencing standard. The development of the application system utilises the 
PHP programming language, alongside JavaScript, jQuery and MySQL database. The following Figure 1 
shows the main interface of the CiteGuru mobile application. 
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Figure 1. The interface of the CiteGuru application 
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CiteGuru application was targeted at university students and academicians that actively involved 
with the publication of refereed articles and manuscripts. Among the special features of the application are i) 
syntax colouring with video assistance, ii) share-it (to share the article), iii) student engagement monitoring 
index (SEM-x)—to monitor user engagement with the application and offer a predictive solution, and iv) APA 
Competency Mapping. The application can be accessed through Google Play or via Progressive Web App. 
Besides, the framework of CiteGuru can be expanded to various other applications such as a dictionary and 
knowledge-based platform. 


3. METHOD 

This study was conducted in quantitative nature. A research instrument was developed by adopting the 
system usability scale (SUS) instrument of [11], [12]. According to Bangor et al. [12], the SUS instrument 
provided an effective way to measure the usability of a variety of services and products. SUS also has been used 
by a variety of research projects, as well as industrial evaluations [11], [13]. Besides SUS also helps researchers 
to collect users’ perceptive measurements on product usability more quickly and easily [12], [14]. 

The research instrument was developed in 2 sections. The first section collects the demographic 
details of the respondent while the second section collects respondents’ perception of the usability of the 
information system. The selection of respondents is based on several criteria: i) has advanced knowledge of 
library information systems, ii) has working experience of more than 10 years, and iii) have conducted 
training on any type of information system. A total of 23 respondents was selected for the purpose of the 
study from various educational background and expertise. 

Next, the respondents were given a short briefing via Google Meet. The purpose of the briefing 
session is to i) explained the purpose of the research, ii) discussed the SUS criteria that will be used for the 
study, and iii) introduced the features and architecture of the CiteGuru application. Data collection was then 
conducted via Google Form to the selected respondents and the finding is analysed using statistical package 
for social sciences (SPSS) version 26. 


4. RESULTS AND DISCUSSION 

The following subsection shows the result of the study: i) We explained the demographic profiles of 
the respondents; ii) The descriptive assessment of usability components was highlighted. Since the SUS 
instrument is mixed between positive and negative statements, reverse coding was performed on the negative 
statements; iii) We tested the demographic profiles (sector, education, and gender) on the usability 
assessment using analysis of variances (ANOVA) and independent sample T-Test; and iv) Usability 
assessment was assessed using the SUS scoring based on the work of [11], [12]. 


4.1. Demographic profiles 

Table 1 shows the demographic profiles of the respondents. A total of 23 expert’s respondents were 
selected for the purpose of the study. Most of the respondents are female (N=15 or 65.2%) while males are 
represented by 8 respondents or 34.8% of the total sampling. In relation to sectors, more than half of the 
respondents work as a librarian (N=14 or 60.9%), followed by academician (N=8 or 34.8%), and industry- 
related (N=1 or 4.3%). The majority of respondents have a working experience of more than 10 years (N=22 
or 95.7%), while one respondent has a working experience of more than 20 years. In regards to the 
educational background, more than half of the respondents (N=18 or 78.3%), followed by bachelor’s degree 
(N=4 or 17.4%) and doctoral (N=1 or 4.3%). 


4.2. Descriptive 

The following Table 2 shows the descriptive analysis of the study. Based on the mean value, the 
highest statement as rated by respondents are ‘I thought the system was easy to use’, followed by ‘I would 
imagine that most people would learn to use this system very quickly’, and ‘I felt very confident using the 
system’. To interpret the data, the positive statements are grouped as USAI, USA3, USA5, USA7, and 
USA9. The overall mean of 4.33 indicates a tendency towards a positive response among the respondents. In 
regard to the negative statement, the combined mean value of USA2, USA4, USA6, USA8, and USA10 is 
1.74, indicating a tendency towards negative responses; thus, justifying the assumption of the usability 
assessment for the application. 


4.3. ANOVA and independent sample T-Test 

Next, a one-way ANOVA was conducted between the demographic profiles (sectors, education, and 
gender) towards the usability assessment data. Table 3 shows the result of ANOVA conducted using SPSS 
version 26. The result indicating: i) There is no significant different between three level of sectors 
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[F(2, 20) =0.899, p=0.423] and ii) There is no significant different between three level of education 
[F(2, 20) =1.772, p=0.196]. 


Table 1. Demographic 


Item Sub-Item Total % 
Gender Male 8 34.8 
Female 15 65.2 
Sector Librarian 14 60.9 
Academician 8 34.8 
Industry 1 43 
Working Experience 10 to 20 years 22 95.7 
More than 20 years 1 43 
Education PhD 1 43 
Master 18 78.3 
Degree 4 17.4 


Table 2. Descriptive 


Item Item Mean Std. Dev. _ Mod 
USAI I think that I would like to use this system frequently 4.22 0.74 4 
USA2 I found the system unnecessarily complex 1.91 0.90 2 
USA3 I thought the system was easy to use 4.52 0.51 5 
USA4 I think that I would need the support of a technical person to be able to use this system 1.70 0.88 2 
USAS I found the various functions in this system were well-integrated 4.26 0.62 4 
USA6 I thought there was too much inconsistency in this system 1.87 0.81 2 
USA7 I would imagine that most people would learn to use this system very quickly 4.35 0.65 4 
USA8 _ I found the system very cumbersome to use 1.57 0.59 2 
USA9 I felt very confident using the system 4.30 0.63 4 
USA10___I needed to learn a lot of things before I could get going with this system 1.65 0.98 1 


Table 3. ANOVA 


Independent _ Dependent Sum of squares df | Mean square F Sig. 
Sector Usability | Between groups 0.357 2 0.179 0.899 0.423 
Within groups 3.972 20 0.199 
Total 4.330 22 
Education Between groups 0.652 2 0.326 1.772 0.196 
Within groups 3.678 20 0.184 
Total 4.330 22 


An independent sample T-Test was conducted between gender and usability. The following Table 4 
shows the independent sample T-Test between gender (male and female) and usability. The test was 
conducted to compare the influence of gender on the usability perception among the selected respondents. 
The result show that there are no significant differences in the score between male (M=3.08, SD=0.36) and 
female (M=3.27, SD=0.49) conditions; t (18.73) = -1.014, p=0.323. The test result suggests that different 
kind of genders does not influence or change the usability result of the study. 


Table 4. Independent sample T-Test 


Variable F Sig. df Sig. (2-tailed) 
Gender Equal variances assumed 1.431 0.245 21 0.368 
Equal variances not assumed 18.734 0.323 


4.4. SUS scoring 

According to Brooke [11], the SUS score is range from 0 to 100. The items are not meaningful by 
itself; meaning that the positive pole and negative pole must be reported altogether. The SUS scoring is 
calculated: i) items 1,3,5,7,9 will minus | from the scale position, ii) items 2,4,6,8, and 10 is calculated by 5 
minus the scale position, iii) All items will be added to get a total score, and iv) the total score will be 
multiplied by 2.5 to obtain the total usability score. The following Table 5 shows the result of usability 
assessment based on three usability acceptance rates: i) Acceptable (70% to 100%), ii) Marginal Acceptance 
(50% to 69%), and iii) Not Acceptable (0% to 49%). The result from Table 5 shows that most of the experts 
rate the application usability assessment as Acceptable (N=22 or 95.6%) and Marginally Accepted 
(N=1 or 4.34%). No expert rates the application as not acceptable, thus confirming the usability of the 
application. The following Table 6 shows the distribution of SUS scores among 23 experts selected for the 
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study. More than half of the experts gave marks more than 70% for the usability assessment of the 
application. The highest marks given is 100% (N=4), while the lowest marks given is 60% (or N=1). 


Table 5. Acceptable ranges 


Scale Total % 
Acceptable (70-100) 22 95.6 
Marginal (50-69) 1 4.34 
Not Acceptable (0-49) 0 0 


Table 6. Acceptable ranges 


# SUS Score % Usability Indicator 
1 33 82.5 Acceptable 
2 31 715 Acceptable 
3 38 95 Acceptable 
4 30 715 Acceptable 
5 40 100 Acceptable 
6 31 77.5 Acceptable 
7 36 90 Acceptable 
8 31 TAS Acceptable 
9 30 75 Acceptable 
10 29 72.5 Acceptable 
11 40 100 Acceptable 
12 30 715 Acceptable 
13 40 100 Acceptable 
14 29 72.5 Acceptable 
15 34 85 Acceptable 
16 31 TS Acceptable 
17 34 85 Acceptable 
18 30 75 Acceptable 
19 33 82.5 Acceptable 
20 35 87.5 Acceptable 
21 29 72.5 Acceptable 
22 40 100 Acceptable 
23 24 60 Marginally 


4.5. Discussion on results 

Demographic profiles have long become a debate among the scholars. Several researchers found a 
positive correlation and significant differences between demographic profiles with the antecedent and dependent 
variables [15]—[21]. For example, Matovic et al. [18] found that female respondents are more satisfied with their 
basic salary and work-related benefits. Besides, age also plays an important role as motivational impacts 
especially to women over the age of 45. On the other hand, the work of Rosman et al. [5] found out that 
different fields of study contributed toward a different level of user engagement with the digital library—in 
which social science recorded a high level of user engagement while science and technology recorded a low to 
moderate level of engagement. However, in the context of usability assessment, much focus is given to the 
assessment of the product rather than the effect of demographic profiles on the usability assessment. The result 
of our study extends the previous similar works by assessing the effect of demographic profiles on the usability 
assessment. Our study shows that different demographic profiles (sector, education, and gender) do not have a 
significant difference over the usability assessment. This can be attributed to the profiling of the respondents—in 
which most respondents have experience over than 10 years in the related fields of library and information 
science (LIS), as mentioned by several previous research [22]-[24]. Besides, most of the respondents have 
previous knowledge on the citation software using either EndNote or Mendeley, thus it helps them to relate the 
interface of the application and its usage. 

On the other hand, the result of usability analysis using SUS score indicating the acceptance of the 
proposed application software as suggested by several usability testing research such as [11], [12], 
Upitis et al. [25], and Rosman et al. [26]. Our result indicates that several criteria must be given utmost 
importance in the development of an information system, such as special features, user-friendliness, proper 
guidance, and training. Moreover, the future study must also consider the aspect of end-user in the 
development of an information system. Additionally, we utilise the SUS score to determine the usability of 
the application. A future study might look upon other usability assessments such as [26]—[28]. 


5. CONCLUSION 
The study was conducted to answer two research objectives: first, to investigate the roles and 
usability assessment of CiteGuru application as a mobile solution for improving learners’ skills, ability, and 
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knowledge on referencing, and second, to investigate the roles of demographic profiles on the usability 
perception among the respondents. To answer both objectives, we conducted a quantitative study involving 
23 experts. Data was then analysed and imported into SPSS version 26. To answer the first objective, we 
adopted the SUS. Data were recorded based on positive and negative perceptions. Our findings indicate that 
the experts rate the CiteGuru application as Acceptable. In relation to the second objective, we use SPSS to 
determine the effect of demographic profiles (sector, education, and gender) on the usability assessment of 
the expert. Two tests were conducted: ANOVA and independent sample T-test. Results show that all three 
conditions (sector, education, and gender) do not have any influences on the usability assessment. This study, 
however, is not without a limitation. First, we collect the data from a small number of respondents. Although 
sufficient in the context of the usability study, we suggest future studies to involve more diverse respondents, 
especially the end-user that will be using the application. Second, this study looks upon the effect of 
demographic profiles on the usability assessment. We suggest future studies to develop theoretical modelling 
on the determinants and impacts of usability assessment especially concerning human-computer interaction 
and user engagement. 
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